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ABSTRACT 

This research considers whether a person’s demographic and experiential 
attributes play a significant role in how they perceive the presence or absence of hazards 
in a given situation. The goal of the research is to show that participants with enlisted 
military experience, prior to being commissioned as a junior officer, would be more 
successful at identifying the hazards presented in military scenarios than those who had 
only been trained on the process via their pre-commissioning and initial entry courses of 
instruction. The research study involves the use of two surveys with realistic military 
scenarios including both Foot March and Maintenance scenarios. The data collected from 
the surveys was analyzed using data mining techniques, in particular Nearest Neighbor 
(NN) algorithm and Logistic Regression Model (LRM). NN determines how similar a 
participant’s case is to an expert case and LRM analyzes the outputs in a way that allows 
us to see if any of the seven experiential and demographic attributes considered had a 
significant impact on a participant’s ability to perform well on the assessment. While the 
results did not conclusively prove that experience or other demographic attributes had a 
statistically significant impact on a participant’s overall performance, the results did 
suggest that the idea that those same attributes do not have an impact cannot be rejected. 
This research could provide useful feedback to the U.S. Army on the way they train and 
educate junior officers on their Risk Management process. 
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1. Introduction 
1.1 Background 

Risk management (also referred to as RM) is defined by the U.S. Army as ‘the 
process of identifying, assessing, and controlling risks arising from operational factors 
and making decisions that balance risk cost with mission benefits’ [1], In 1998 the Army 
introduced the first doctrinal publication on managing risk, the now obsolete Field 
Manual 100-14, in recognition of the need to standardize a methodology for identifying, 
quantifying, and mitigating the risks associated with training and combat activities [1]. 
Though the specifics and scope of this process have changed over the intervening years, 
the overarching goal of RM is still to manage the inherent risk as well as eliminate all 
unnecessary risk in all Army activities. While the Army has done much to improve this 
process with a goal toward making it more standardized and, as a consequence, less 
subjective, it has not significantly addressed a fundamental issue of this subjectivity, 
specifically why different individuals, presented with the same situation, perceive hazards 
differently. Conventional wisdom within the Army would argue that experience is the 
key to success in RM. Admittedly, having executed a specific task once or a number of 
times before can provide a risk manager a degree of perspective and historical knowledge 
that can certainly be leveraged when considering risk in similar future situations. But 
does this experience or any other demographic factors outside the scope of this task- 
specific experience shape an individual’s perception of risk and ultimately their success 
in identifying hazards? 
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The aim of this thesis is to address these issues by analyzing the results of two 
scenario-based risk assessments completed by a population of newly-commissioned 
Infantry Officers who were attending the U.S. Army Infantry Basic Officers Leadership 
Course at Fort Benning, GA. 

1.2 Research Question 

The work done in this thesis aimed to answer the following research question: 
“Would the demographic attributes of the participants have a significant impact on their 
overall ability to correctly identify the hazards presented”? The hypotheses developed 
from this question are: 

Ho: Demographic attributes do not have an impact on the overall performance of the 
participants. 

Hi: Demographic attributes do have an impact on the overall performance of the 
participants. 

To validate or refute the alternative hypothesis, data mining techniques such as 
Nearest Neighbor (NN) algorithm and logistic regression techniques are used. NN aims 
to determine a Degree of similarity (DoS) between an expert case and each participant’s 
input. The DoS or scores were then analyzed against seven different demographic 
attributes of each participant using a Logistic Regression Model (LRM) with the intent of 
identifying any of the attributes that had a significant statistical impact on the 
participant’s performance. 



IMUIMUMMIMIllBMUIIimilllUllllimmimiiWIIlMIMMIIHIUlIIIIJI 
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1.3 An overview of the Army RM process 

The Army describes RM as the process for helping organizations and individuals 
make informed decisions to reduce or offset risk [1]. The current model is based on four 
underlying principles: integrate RM into all phases of missions and operations, make risk 
decisions at the appropriate level, accept no unnecessary risk, and apply RM cyclically 
and continuously. These four principles drive the cyclical and continuous five-step 
model as illustrated in Figure 1 below. Although each of these steps plays an important 
role in the process, for the purposes of this study, the scope of research is limited to the 
first step of the Army’s JIM process, Identify the Hazards. This is because without the 
ability to accurately identify the hazards present in the situation, a RM user cannot 
effectively assess or mitigate them in the subsequent steps of the process. In other 
words, you can’t assess what you do not know is there. As Rotar and Kozar, stated in 
their work on the mechanics of an RM process, “The value of a risk management process 
is reduced without a clear understanding of the sources of risk and how they should be 
responded to” [2], The U.S. Army’s RM process, depicted below in Figure 1, is a five- 
step process which consists of identifying the hazards, assessing those hazards, 
developing controls and making risk decisions, implementing controls, and supervising 
and evaluating throughout the execution of the event. 
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Step 1 

Identify the 
Hazards 



Step 2 

Assess the Hazards 



Step 3 

Develop Controls 
and Make Risk 
Decisions 



Figure 1: The Army Risk Management Process [1] 

To assist risk managers in the first step of the RM process, the Army doctrine 
employs a series of considerations known as the mission variables. The mission 
variables are a construct that are taught as a tool or framework with which to identify and 
assess hazards present in a given situation. They consist of the Mission, the Enemy, the 
Troops and Equipment available for the task, the Time available to complete the task, the 
Terrain and weather in which the task will be executed, and any considerations involving 
the presence of Civilians on the battlefield. These variables are represented by the 
acronym METT-TC. During Step 1 of the RM process, each of the variables is 
considered as a possible hazard source. A list of considerations for each variable is 
provided by [1], an excerpt of which is illustrated below in Table 1. Note that this list is 
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not meant to be all inclusive, but is instead to be used as the framework with which to 
initiate the process. 

Table 1 : Mission Variables and Considerations [1 ] 



Mission Variable 


Consideration(s) 


Mission 


Army leaders look for hazards associated with the complexity of plans and 
orders from higher headquarters, such as a particularly complex scheme of 
maneuver. 


Enemy 


Commanders look for enemy presence or capabilities that pose hazards 
and risks to operations. They ask what the enemy or outside influences 
could do to defeat or spoil an operation. 


Terrain & 
weather 


Army leaders should include the aspect of terrain. Common terrain 
hazards are elevation, altitude, road size and surfaces, curves, grades, and 
traffic density. Weather can also create specific hazards and risks. 
Common weather hazards are cold, ice, snow, rain, fog, heat, humidity, 
wind, dust, visibility, and illumination. 


Troops & 
equipment 
available 


The variable troops is used to identify hazards and risks associated with 
the level of training, staffing, and equipment maintenance and condition. 
This factor also includes hazards related to morale, availability of 
supplies, and services. Moreover, it includes hazards related to the 
physical and emotional health of each individual. 


Time available 


Subordinate commands need adequate planning and preparation time to 
develop and implement controls. Insufficient time for planning or 
preparation may lead to accepting greater risk. (For activities not directly 
related to operations, insufficient planning or preparation time usually 
results from haste rather than availability of time.) 


Civilians on the 
Battlefield 


The variable civil consideration expands the consideration of hazards and 
risks to include those that a tactical task may pose to the civilian populace 
and noncombatants in the operational area. It includes the critical 
requirement to protect civilians. The objective is to reduce collateral 
damage to civilians and noncombatants 



These variables provide the baseline for a standardized, comprehensive method 
for identifying risks which is taught throughout the U.S. Army and were used in this 
study as a means of analyzing the participant population’s input into the study, which is 
discussed in detail in Chapter 4. 



The remainder of this thesis provides a detailed discussion of this effort. The next 
chapter provides a survey of existing research related to RM as well as some past and 
current efforts by researchers to provide RM systems in a variety of fields and industries. 
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Chapter 3 provides an overview of the research design and methodology including the 
recruitment strategy and data collection methods. Chapter 4 discusses the research tools 
used throughout the study. Chapter 5 highlights the characteristics and use of the LRM to 
produce results. Chapter 6 provides an analysis of the results. Finally, Chapter 7 
provides discussion of the study, including limitations the study dealt with, and 
recommendations for future research in this area. 
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2. Related Work 

Risk management has been an active research area in the past few decades. 
Despite the importance of this field in many domains such as insurance, investing, and 
information assurance, and others, the specific act of identifying risks in the U.S. Army 
has been under-examined. A survey of current RM literature and research shows that a 
variety of tools exist which intend to optimize the RM process by leveraging expert input 
to establish rules or parameters and then employ algorithmic or other functions modeled 
on how humans think in order to manage risk. Some of the more common of these RM 
processes employ such techniques as data mining, NN algorithms, regression, or expert 
systems (ES) such as neural networks, fuzzy logic, and even gaming systems to achieve 
their purpose. 

Some leaders in the RM field have proposed various ESs to provide the expertise 
and objectivity required to effectively identify hazards and manage risk (some of the 
more successful of which are discussed in the second part of this section). However, a 
comprehensive study of existing literature relating to current RM processes underscores 
the idea that bias or subjectivity can play a significant role in how an RM user perceives 
risk. But is this necessarily bad? What if this subjectivity were based on experience or 
some other sources that lent itself to a better outcome? Unfortunately, little quantitative 
research is available that approaches, from the perspective mentioned above, the ability 
of an individual to successfully identify hazard. Indeed, in his perspective on quantitative 
risk assessment, George E. Apostolakis points out that “While it is relatively easy to 
ascribe an accident that has occurred to a bad safety culture, the fact that defining 
indicators of a good or bad safety culture in a predictive way remains elusive" [3]. The 
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remainder of this section is dedicated to a review of data mining and ES designed to 
enable RM, as well as a survey of the current literature on the RM in the U.S. Army. 

2.1 Existing systems for risk management 

In [4], Bertrand Laporte proposed a data mining-based RM tool intended to assist 
customs agents responsible for inspecting baggage and personnel wanting to enter a 
country with accurately assessing who and what present the greatest hazards. He 
presented his idea based on the premise that by capturing specific data (some of which 
includes the contents of such documents and activities as verification certificates, detailed 
declarations, and the results of inspections for a specified reference period) and then 
effectively managing and mining that data, the user can establish accurate risk profiles 
based on “statistical regularities” that result. 

Similar to LaPorte’s concept, Maria Fernanda D’Atri, Dario Rodriguez, and 
Ramon Garcia-Martinez presented a paper in which they employ data mining as a means 
of exploiting information based on intelligent systems to improve and optimize risk 
models used in the gas pipeline industry [5], 

As mentioned previously, another popular way to counter the subjectivity inherent 
in so many RM systems (essentially an effort to level the playing field between those 
with experience in RM and those without) is to develop an ES that, to the extent possible, 
relies on expert input and logical inference to determine the hazards present and overall 
risk level associated with a given activity. A study of existing techniques employing ES 
to address and manage risk in a comprehensive manner reveals several credible proposals 
and existing applications in areas including supply chain management, financial 
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management, insurance, and information assurance. An overview of each of these 
systems is provided in the following paragraphs. 

In their paper [6] on quantifying risks in those supply chains, Samvedi, Jain, and 
Chan discuss the fact that processes for managing risk in supply chains are not immune to 
the same challenges that RM systems in nearly any field face. Indeed, they point out that 
“The major hindrance in analyzing risks comes from the fact that there is a lot of 
subjectivity involved”. Their proposed use of an analytical hierarchy process (AHP) and 
a fuzzy technique for order preference by similarity to the ideal solution (TOPSIS) both 
recognized the subjectivity involved in the process and produced crisp output, ultimately 
producing a risk index and overall risk assessment which can be used to help decision 
makers deal with the risk present. 

Baesens, et al. attempted to tackle risk in the financial industry by proposing the 
use of neural network rule extraction and decision tables as a viable means to evaluate the 
risk associated with extending credit to a given applicant in [7]. Though early attempts at 
using neural networks to evaluate credit risk were successful in evaluating the risk, the 
use of a neural network alone prevented the user from knowing how the classification 
was being made. This proved to be an issue as it prevented the financial institutions from 
meeting their legal obligations to justify why an individual’s credit request had been 
denied. Baesens and his colleagues showed that the use of an appropriate neural network 
(in their case, either Neurorule or Trepan), combined with a decision table for 
visualization, can result in successful financial risk evaluation. The decision table served 
as a visualization tool that represented the neural network rule extraction process in a 
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visual and understandable way, thus making it usable to the decision maker in justifying 
their decisions. 

Like evaluating credit risk, selecting the appropriate mix of stock, bonds, etc that 
match an investor’s acceptable level of risk can have an enormous impact on the success 
or failure of a financial manager. In [8] Shane, Fry, and Toro wrote about their 
development of a decision support system that uses two knowledge bases, a database, and 
a gaming system to effectively and efficiently match the appropriate portfolio to the 
investor. The first knowledge base, known as Investment Suitability addresses the 
appropriateness of an investment type with the personal attributes of the investor. A 
gaming system is used to determine overall investor risk tolerance. The second 
knowledge base. Economic Conditions , then analyzes and applies the effects of current 
market conditions against the Investment Suitability results. Finally, a Portfolio 
Preference database is consulted which contains historical portfolio mixes made by the 
advisor to see if any suitable options already exist. The output of these systems is then 
used to provide an informed recommendation to the investor on his or her optimal 
investment portfolio mix. 

In [9], their paper studying the use of ES for RM in the insurance industry, Meyer, 
et al. conducted a study of two large insurance firms, John Hancock and Lincoln 
National, and their successful efforts in developing effective ES for decision support of 
underwriting insurance policies. In the case of John Hancock, which specializes in 
directly offering insurance policies to a large number of consumers, the use of an ES was 
focused on improving efficiency by employing a specially-developed set of heuristics 
which would allow a large number of more routine cases (based on the applicant’s 
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occupation, financial issues, medical conditions, etc) to be automatically approved, 
thereby reducing the workload of human underwriting experts to only those cases that fell 
outside of these norms. In the case of Lincoln National, the ES was designed to more 
efficiently combine what the authors classify as the procedural knowledge of the 
underwriting expert with the more theory-based inference done by the medical experts 
and actuaries employed by the company. 

Fenz, Eckelhart, and Neubauer [10] developed an Automated Risk and Utility 
Management (AURUM) system to provide comprehensive RM support to those working 
in the area of Information Assurance (1A). Their system uses a Bayesian network to 
calculate threat probabilities, a risk determination, risk control identification, and 
evaluation. This process is intended to support decision makers in gaining a better 
understanding of the risks they face while also providing him or her with what can be 
achieved in terms of addressing the risks in relation to opportunity costs to other efforts 
or objectives. 

While each of the ESs discussed above do indeed make strides in addressing the 
subjectivity issue associated with RM, they are mostly narrow in scope and serve to 
quantify the risks presented by known hazards. In other words, the ES themselves may 
analyze the hazards, but do so only once the experts have identified them. While 
admittedly useful to the individual risk manager, they fall short of providing the larger 
organization with any sort of understanding as to why the assessor or expert identified 
certain hazards when others did not. They are not, by design, analysis tools for use in 
addressing the challenges this study attempted to deal with. Until they can understand 
the root cause of the subjectivity involved in RM, organizations may never be able to 
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attain the lofty goal of training and embedding risk management into the actual culture or 
fabric of the organization [11]. 

2.2 Current research on Risk Management in the U.S. Army 

The U.S. Army provides a large number of publications in which each addresses 
RM and provide a good deal of detail relating to the process of RM, but do little to 
discuss or more importantly quantify any factors that might lead an individual to 
recognize, or fail to recognize, hazards in a given situation. These include, but are not 
limited to [1], [12], [13], [14], and [15], The U.S. Army Safety Center publishes monthly 
and annual Army-wide accident statistics which can be used for trend analysis, but the 
researcher could not find any that delved into any demographic or psychographic data 
relating to the individual conducting the risk assessment for the event that led to the 
accident or injury in question [16], 

Chris W. Johnson has authored two excellent works relating to RM in the 
military. His paper [17] discusses the U.S. Army’s various RM processes. He goes as far 
as to point out that given a number of the Army’s current processes (in this case relating 
specifically to aviation risk), “There are few guarantees that different personnel will 
identify similar hazards for the same mission elements”. In his other paper [18], he 
discusses the idea that members of the military may tend to be more risk-seeking or 
tolerant. Dr. Johnson posits that “There seems to be very little direct evidence today that 
CRM (Composite Risk Management) techniques will be able to compensate for the risk 
preference biases that are often seen in military personnel”. It can be argued that both of 
these points speak to the idea that when coupled with the uniformity provided by an 
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effective RM process, subjectivity, depending on its source or cause, can either enhance 
or degrade the usefulness of that RM process. 

In their work about examining the implementation of RM approaches in military 
operations, Liwang, Ericson, and Bang note that as powerful of a tool RM is, it can only 
be successful with an understood and shared definition of risk [19]. Unfortunately, they 
do not go as far as to delve into whether or not previous experience in considering risk 
plays a role in how one perceives it. 

Kamperis, et al provide an interesting and informative analysis of quantitative and 
qualitative techniques for risk assessment in [20], but ultimately acknowledge that the 
common weakness of both techniques is that the assumptions used by those employing 
them can be highly subjective. While true, one could still argue that although that 
subjectivity is present, experience or some other demographic attribute associated with 
the assessor might provide a more informed version of subjectivity, which could actually 
result in a more accurate assessment of risk. 

Some recent research and publications relating to RM in the civilian sector have 
focused on attempts to refine RM processes, with the idea that the simpler the process is 
to use, the more likely a supervisor or worker is to actually use the RM system as 
opposed to relying solely on their “wits” or experience. For example, in [21] Pinheiro, 
Cranor, and Anderson discuss RM in the oil and gas production industry and propose a 
modified approach to the legacy processes that includes a “simplified approach for 
determining risk”. The approach may be simplified, but if the designer does not attempt 
to understand why some users are successful and others aren’t, then there is still a 
knowledge gap that will keep the system from being as effective as possible. The next 
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chapter introduces the reader to the research tools used to address the issues discussed 
above. 
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3. Research Design & Methodology 
3.1 Research design 

As described in the previous chapters, although there is abundant research on RM 
in different areas, such as the commercial insurance, banking, and investment sectors, it 
seems that research on RM in the U.S. Army has been under-examined. The question 
addressed in this thesis is: Would the demographic attributes of the participants have a 
significant impact on their overall ability to successfully identify the hazards presented? 
The research question leads to the following hypotheses: 

Ho: Demographic attributes do not have an impact on the overall performance of the 
participants. 

H,: Demographic attributes do have an impact on the overall performance of the 
participants. 

To address the research question, data was collected using a case study model featuring a 
within-group design in which each participant completed risk assessments based on two 
scenarios: a foot march and maintenance on a vehicle-based version. A qualitative study 
was conducted using RM worksheets with individual cases within the study consisting of 
the participants themselves (i.e. their biographical and experiential data), as well as their 
responses to the scenarios. The researcher employed theoretical replication by using 
comparable cases (all 2 nd Lieutenants with recent RM training) to generate different 
results which were explained by differences between each case (i.e. their biographical 
and experiential data). 
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In order to prevent participant fatigue during the study, all participants were 
given one week to complete the assignment. The approximate duration required to 
complete the assignment was two hours. Additionally, the researcher attempted to 
minimize any bias resulting from learning effects by providing two separate scenarios for 
participants to complete, as well as by not dictating the order in which they should 
complete them. 

3.2 Research methodology 

As was discussed in the introduction to this thesis, there is little research available 
that directly addresses the question of why different individuals tend to identify different 
hazards, even when viewing the same situation. The researcher aimed to address this 
issue by providing a group of participants with identical risk scenarios, having them 
conduct their individual risk assessments on their scenarios, and then analyzing their 
results to determine what demographic attributes might have a statistically significant 
impact on the participants’ performance. 

3.2. 1 Recruitment strategy 

Participants in this research consisted of U.S. Army and Marine officers, all in the grade 
of 2 nd Lieutenant, who were attending the Infantry Basic Officer Leadership Course 
(IBOLC) through the U.S. Army Maneuver Center of Excellence (MCoE) and the 
Infantry School at Fort Benning, GA. IBOLC serves as initial entry training for all newly 
commissioned officers in the Infantry branch. This is the first training that 2 nd 
Lieutenants receive after earning their commissions from either a service academy 
(USMA), the Reserve Officers Training Corps (ROTC), or the Officer Candidate School 
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(OCS). The goal of 1BOLC is to educate, train, and inspire Infantry Lieutenants so that 
upon IBOLC graduation, they demonstrate the competence, confidence, physical and 
mental toughness, and moral/ethical fiber necessary to lead platoons in any operational 
environment [22], Immediately prior to participating in the study, participants received a 
formal block of instruction intended to prepare them for a subsequent homework 
assignment requiring each student to read through two realistic scenarios and then 
complete the RM process on each scenario using a DA Form 7566, the Army’s 
Composite Risk Management Worksheet. 

Participants in this study were contacted through their IBOLC chain of command 
and asked to participate on a voluntary basis. It is important to note that while the 
assignment to complete the RM scenarios was administered by the participants’ 
instructors as part of their normal classroom Risk Management curriculum, participation 
in the study was voluntary. In addition to the assigned homework, those who chose to 
participate in the study received and completed a biographical questionnaire, as well as 
an Informed Consent Form. 

3.2.2 Data collection 

Data considered for the study consisted of responses derived from risk assessments 
completed by study participants in response to a realistic military training scenario 
provided by the researcher. The construct for assessing risks and identifying any hazards 
presented throughout the scenarios was organized into four categories, each of which 
were derived from portions of the U.S. Army’s mission variables, which are comprised of 
Mission, Enemy, Terrain, Troops and equipment available. Time available, and Civilians 
on the battlefield, also known as METT-TC. The reader should note that only four of the 
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six mission variables were considered. This is because the scenarios that drove the input 
for the system were based on training situations and as such did not require the 
participant to consider the variables of Enemy or Civilians on the battlefield. Therefore, 
the researcher used a condensed version of METT-TC, specifically one that accounted for 
the elements of Mission, Terrain, Troops and equipment available, and Time available, or 
MTT-T. Each of these four categories contained a series of binary attributes that 
represented a possible hazard condition within that category. The condition of each was 
such that it was either identified as present or not present. If a hazard was identified, the 
researcher categorized it into one of the four categories discussed above. A fifth category 
of data collected represented the demographic attributes associated with each participant, 
reflecting the biographical and experiential data associated with the participant 
represented. 
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4. Research Tools 

A data analysis model has been proposed to analyze the data collected from the 
surveys as shown in Figure 2. The researcher used data mining via a Nearest neighbor 
(NN) in order to calculate the Degree of similarity (DoS) between each participant’s 
input and an expert case. These DoS, or scores, were later used for identifying 
subgroups of a participant population who demonstrated similar tendencies in identifying 
or failing to identify hazards as part of an RM process, and then analyzing those 
subgroups in order to identify any trends based on characteristics (i.e. demographic , 
experiential, etc) present within the similarly responding groups. 

The individual records were input into a NN algorithm in the form of individual 
cases and were compared against an expert case in order to calculate the similarity 
between them. A NN process stores all available cases to project a numerical target 
based on the similarity measure or distance function [23]. These distance functions 
became the DoS for each case. Demographic attributes (with the exception of name) 
were not introduced into the NN algorithm — they existed as a metric so that individual 
records within data clusters could later be extracted for use. The output was then grouped 
according to DoS parity. Selected clusters or groupings were then analyzed to determine 
any common trends based on the demographic attributes associated with the member 



individuals. 
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Figure 2: Data analysis model 



4.1 Nearest Neighbor (NN) algorithm 

The system leveraged the NN algorithm offered by the off-the-shelf myCBR tool 
[24] to calculate the DoS between each participant case and an expert case provided by 
the researcher. myCBR is an open-source similarity-based retrieval tool that can model 
and test similarity measures between cases [25]. These inputs, each of which 
constituted one member case, were derived from the responses of 72 separate study 
participants in response to the two different risk management scenarios. The first, 
Scenario 1 , was based on a U.S. Army infantry platoon preparing for and conducting a 
tactical foot march, or ground-based movement, for training. The second. Scenario 2, 
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was based on a similar platoon conducting vehicle maintenance and subsequent vehicular 
movement to a railroad loading area to prepare the vehicles for shipment by rail. Using 
current Army doctrine, including the modified MTT-T construct and the Army’s RM 
process as a foundation, the researcher identified fourteen potential hazards shared by 
these scenarios and then adjudicated the presence or absence of these hazards in each. 
The researcher determined, for the purpose of the expert case, that in Scenario 1, there 
were nine of 14 potential hazards present. In Scenario 2, there were 11 of 14 present. 
These 14 potential hazards became the hazard attributes associated with each scenario 
and the expert case. These hazard attributes, discussed in detail in the next section, were 
assigned either a ‘Yes’ if they represented a hazard that was present in the scenario or a 
‘No’ if they did not. The values associated with the expert case were introduced by 
means of the query function of NN. The software then compared each case in the case 
base to the expert case and returned a corresponding similarity value for each case. The 
similarity values were then used to determine groupings of similar responses, which the 
researcher used to identify any experiential or biographical trends amongst the clustered 
participants as part of a later phase of the study. 

4. 1. 1 Attributes in NN algorithm 

There were a total of 14 hazard identification attributes and seven biographical, 
experiential, and administrative demographic attributes considered in this study. 
Biographical attributes were recorded for participant identification and later data analysis, 
but (with the exception of each participant’s name) were not considered in relation to the 
NN algorithm in order to minimize skewing in the DoS calculation. Table 2 below 
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provides a description of each hazard attribute used in the system, listed under its 
applicable MTT-T category (designated by the shaded rows). 



Table 2: Hazard Identification Attribute Descriptions 



Attribute 


Description 


Mission 


C2 Span of Ctrl 


Can leaders sufficiently control organization throughout movement? 


Mission Complexity 


Does level of mission complexity present a challenge or hazard? 


SOP Availability 


Does the platoon have a well understood Standard Operating Procedure 
(SOP) for this task? 


Guidance from HQ 


Did higher headquarters provide sufficient level of detailed guidance? 


Terrain & Weather 


Weather 


Does the predicted weather pose a hazard to the platoon throughout the 
mission? 


Route 


Does the planned route or terrain to be traversed present any hazards? 


Illumination 


Will limited visibility (fog, darkness, etc) impact mission safety? 


Traffic 


Does traffic (civilian or military) pose a hazard to the platoon throughout 
the mission? 


Troops & Equipment Available 


PLT Task 

Experience 


Does the platoon as a whole have sufficient experience in this task? 


Condition of 
Soldiers and Equip 


Are soldiers adequately conditioned? 

Do soldiers have appropriate and adequate equipment for task? 


Ldr Experience 


Does the platoon leadership that is present have sufficient experience in 
this task? 


Key Pax 
Availability 


Are all key and critical personnel available to participate in mission? 


Time Available 


Planning Time 
Available 


Does the leadership have sufficient time to plan the mission? 


Time to Complete 
Mission 


Does the platoon have sufficient time to complete the mission safely? 


Biographical Data 


Name 


Participant’s name 
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Hazards were either considered present (as represented by a ‘Yes’) or not present (as 
represented by a ‘No’). Each hazard attribute was assigned the attribute type symbol. The 
value was either a match for the expert value, in which case it was assigned a value of 1, 
or a non-match, in which it was assigned a value of 0. One attribute, name, was assigned 
the attribute type string and assigned an jundefined value in the expert case, which was 
done to allow the attribute to be associated with each case (in order to track which case 
was which), but with minimal skewing of the degree of similarity data that was returned. 
Table 3 below provides a detailed listing of attribute names, types, and values, including 
the ‘Query (Expert) Value’ column which represents the expert value for each attribute in 
each scenario as determined by the researcher for the narrative or scenario provided to 
each participant. The values represented in each column were used to populate the Query 
function of the system for each scenario. 
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Table 3: Listing of the name, type, value range, and query (expert) value for each attribute 
considered. 





Query (Expert) Value 




Attribute 


Type 


Inputs 


Foot march 


Maintenance 


Algorithm 

Assigned 

Values 


C2 Span of Ctrl 


Symbol 


Yes or No 


No 


No 


0 orl 


Condition of Soldiers and 
Equip 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


Guidance from HQ 


Symbol 


Yes or No 


No 


No 


0 orl 


Illumination 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


Key Pax Availability 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


Ldr Experience 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


Mission Complexity 


Symbol 


Yes or No 


No 


No 


0 orl 


Name 


String 


name 


undefined 


undefined 


name 


PLT Task Experience 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


Planning Time Available 


Symbol 


Yes or No 


No 


Yes 


0 orl 


Route 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


SOP Availability 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


Time to Complete 
Mission 


Symbol 


Yes or No 


No 


Yes 


0 orl 


Traffic 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 


Weather 


Symbol 


Yes or No 


Yes 


Yes 


0 orl 



4.1.2. Results for NN algorithm 
4. 1.2.1. Data validation 

To validate the data we have, a sample test was done using 15 participants for Scenario 1 
and ran their inputs in all 14 hazard attributes against the expert case. Same test was 
repeated using 1 5 participants for Scenario 2. All attributes returned acceptable results 
(acceptable being defined as no unknown or undefined values being returned for any 
attribute). The researcher then repeated the tests for both Scenarios 1 and 2 using the 
same inputs and query values to ensure that the system reproduces the same results, given 
the same inputs and parameters. There were no anomalies between the first and second 
rounds of testing for either set of cases or query function. 
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4. 1.2.2 Output from NN algorithm 

The overall DoS were then calculated by comparing the attribute values of each 
individual participant’s data against the expert’s data. The algorithm assigned a DoS 
value to each participant based on the proximity of its attribute values to those of the 
expert data values used in the query. The greater the value of the DoS that the algorithm 
assigned, the greater the number of attribute values that matched between the individual 
case and the expert case. No similarity functions or weights were assigned to any 
attributes or the query values as each hazard attribute was of equal value and the only 
possible outcomes for the symbol attributes was either ‘Yes’ or ‘No’. 

Figure 3 contains a sample screen shot of four cases, including their individual 
values and corresponding DoS based on the Foot March (Scenario 1) and expert values as 
illustrated previously in Table 3. 
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FM (No Bio)33 


FM (No 8ic,40 


FM (Nc Bic/42 


FM (No Bio)41 


Similarity' 


m 


0.86 


0.86 


056 


C2 Span of Ctrl 


Yes 


No 


No 


No 


Condition of Soldiers and Equip 


Yes 


No 


Yes 


Yes 


Guidance from HQ 


No 


No 


No 


No 


Domination 


Yes 


Yes 


No 


Yes 


Key Pax Availability 


Yes 


Yes 


No 


No 


Ldr Experience 


No 


No 


No 


No 


Mission Complexity 


No 


No 


No 


No 


Name 


Lichtfuss 


Garner 


Naber 


Semonis 


PIT Task Experience 


Yes 


No 


Yes 


No 


planning Time Available 


No 


No 


No 


No 


Route 


Yes 


Yes 


Yes 


Yes 


SOP Availability 


Yes 


No 


No 


No 


Time to Complete Mission 


No 


No 


No 


No 


Traffic 


Yes 


Yes 


Yes 


Yes 

* 



Figure 3: Example Case Outputs and Degrees of Similarity 

In the example above Lichtfuss, case FM(No Bio)33, correctly identified 1 2 of the 
1 4 hazards and received a 0.93 DoS, while Gamer, Naber, and Semonis (participant data 
40-42 respectively) correctly identified 10 out of 14 hazards and were assessed a 0.86 
degree of similarity. The four cases above serve as a representative example of the 72 
cases per scenario that were compared using the NN algorithm to compute the degree of 
similarity between the four cases and the expert case. 

In the next chapter, the researcher discusses how the outputs from the NN 
algorithm were analyzed using a Logistic Regression Model to determine the significance 
of any of the demographic attributes on a participant’s score. 
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5. Logistic Regression Model to analyze NN outputs for analysis 

In the previous step, the study data was mined using NN algorithm to determine 
each participant’s performance as compared to an expert case. The researcher employed 
the algorithm to calculate the Degree of similarity (DoS) between the participants’ inputs 
and that of an expert case. The outputs of the NN algorithm (consisting of individual 
DoS) were then collected for analysis using Logistic Regression Model (LRM) using 
XLMiner data mining software to determine if any of the seven biographical or 
experiential attributes had a statistically significant impact on the population’s success in 
identifying hazards. The LRM allows the researcher to establish a relationship between a 
binary outcome variable and a group of predictor variables. It models the logit- 
transformed probability as a linear relationship with the predictor variables and employs 
the equation below. When executed, the logistic regression ofyonxi,..., Xk estimates 
parameter values for p 0 , Pi, • • • , Pk via maximum likelihood method. [26] 

logit(p) = log(p/(l-p))= p 0 + Pi*xl + ... + p k *xk 

Explanation of terms: 

y = binary outcome variable indicating failure/success with 0/1 
p = the probability of y to be 1 , p = prob(y=l ) 

Xj, x k = set of predictor variables 



In this model, y reflected whether the participant had received a “passing” score or not. 
A passing score was defined as having scored above the mean average DoS on both 
scenarios 1 and 2. A non-passing score was defined as a participant failing to achieve a 
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DoS above the mean average on either of the two scenarios or both scenarios. xl,...x7 
were the seven demographic attributes associated with each participant. 

The overall LRM process consisted of three steps: Coding the data for analysis, 
organizing the data for input into the regression model, and finally running the regression 
model. The first two steps are discussed in detail in the following subparagraphs. 

5.1 Data coding 

The biographical and experiential data discussed in Chapter 3 was organized into 
seven areas of consideration. These areas were: platoon assignment, commissioning 
source, academic discipline, if the participant had any prior military service before being 
commissioned (not including pre-commissioning training or education), actual time in 
military service, whether they received a reserve or active duty commission, and number 
of deployments. For analytical purposes, this data was then coded, as illustrated in 
Tables 4 and 5 below. 



Table 4: Biographical Attribute Coding 



Attribute 


Coding Options 


Platoon Assigned to 


1,2, or 3 


Commissioning Source 


Service Academy, ROTC, or OCS 


Prior Service 


Yes or No 


Component 


Active Component or Reserve Component 


Academic Discipline 


0,1 ,2,3,4, or 5 ( *)See table 5 below for explanation 


Time in Service 


< 1 year, 1 -3 years, or >3years 


Number of Deployments 


0, 1, 2, or 3 
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Table 5: Breakdown of participants’ academic majors by discipline 



Discipline 


Code Number 


Majors Included 


No Major 


0 


No major declared 


Physical Science 


1 


Biology 
Chemistry 
Exercise Science 
Geo Science 


Social Sciences 


2 


Political Science 
Psychology 
Criminal Justice 
Sociology 
Languages 
Law 

International / Foreign Relations 
Defense & Strategic Studies 


Business 


3 


Economics 

Business Management 
Culinary Management 


Math/Technology /Engineering 


4 


Mathematics 
Technology 
Engineering 
Operational Research 


History 


5 


History 



5.2 Organizing data for input into the regression model 

Input into the regression model consisted of the values for each of the seven 
demographic attributes for each participant. Output consisted of the dependent variable 
AVG_1. AVG l represented those participants who attained a “passing” score overall. 
Table 6 below extends the previous coding methodology to illustrate the corresponding 
input variables represented in the regression model. The results are discussed in Chapter 



6 of this thesis. 
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Table 6: LRM input and output variables 



Attribute 


Coding Options 


Input Variable 


Platoon Assigned to 


1, 2, or 3 


PLT 1, PLT 2, PLT 3 


Commissioning Source 


Service Academy, ROTC, or OCS 


COM 1, COM 2, COM 3 


Prior Service 


Yes or No 


PSVC 1,PSVC 2 


Component 


Active or Reserve Component 


COMP 1 , COMP 2 


Academic Discipline 


0,1, 2,3 ,4, or 5 


AD 0, AD 1, AD 2, AD 3, 
AD 4, AD 5 


Time in Service 


< 1 year, 1-3 years, or >3years 


TIS 1 , TIS 2, TIS 3 


Number of Deployments 


0, 1 , 2, or 3 


DEPL 0, DEPL 1 , DEPL 2, 
DEPL 3 


Attribute 


Coding Options 


Output Variable 


DoS > mean on both 
scenarios 


Pass 


AVGl 
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6. Results 

This study employed two data mining techniques in an attempt to determine if any 
of the demographic attributes of the participants had a significant impact on their overall 
ability to identify the hazards presented in two military operation-based scenarios. The 
first employed a NN algorithm to determine the DoS between 72 different participant’s 
responses to the two scenarios and an expert case. The second employed logistic 
regression of those outputs to test for a relationship between a binary outcome variable 
and a group of predictor variables. In other words, the logistic regression should 
determine if any of seven demographic attributes associated with the participants had a 
statistically significant impact on their overall success in attempting to match the expert 
case. 

6.1 NN algorithm results 

The NN algorithm was run on the input of each of the two different scenarios. 
The results are below in Figures 4 and 5. In each figure, the X-axis represents the case 
number for each participant (0-71). The Y-axis represents the range of DoS for that 
scenario. Y-values represent each participant’s DoS. The horizontal, dashed line 
represents the mean average DoS for that scenario. 
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Scenario 1 (Footmarch) Results 




♦ Y-Values 



Figure 4: Scenario 1 NN Results 



As seen in Figure 4, the mean average DoS for Scenario 1 was 0.8191861. The high DoS 
was a 0.97 achieved by case number 68. The low DoS was a seven-way tie of 0.73 
registered by case numbers 16, 25, 30, 32, 34, 58, and 61. A total of 48 participants 
scored above the mean average DoS. 
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Scenario 2 (Maintenance) Results 
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Figure 5: Scenario 2 NN Results 

Figure 5 shows the mean average DoS for Scenario 2 was 0.695972. The high DoS was a 
0.86 achieved by case number 3. The low DoS was a three-way tie of 0.58 registered by 
case numbers 33, 61, and 66. A total of 28 participants scored above the mean average 
DoS for this scenario. 



Overall, participants scored better on Scenario 1, which had mean average DoS 
0.1232141 degrees higher than that of Scenario 2. Further, 67% of participants scored 
above the mean average DoS for Scenario 1 while only 39% did the same on Scenario 2. 
Twenty- five participants scored above the mean average DoS on both scenarios. 



mm 



* 
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6.2 Logistic Regression model results 

An LRM using XLMiner was run on the output of the NN algorithm to determine 
the statistical significance of one or more demographic attributes on the scores of the 
participants who scored above average DoS on both scenarios. LRM aimed to compare 
each of seven demographic attributes associated with each participant and their 
associated “passing” or “failing” grade as compared to the mean average DoS from the 
two scenarios. The LRM output results are below in Table 7. Note that the first input 
variable in each input variable category (i.e. PLT1 of the PLT variable category) does 
not appear in the table. This is because the non-visible variables served as the base 
variables by which the others were compared and the odds were calculated. Each of 
these variables was still considered in the calculation, as well as the output and results. 



Table 7: LRM Output Results 



Input 

Variables 


P-Value 


PLT 2 


0.11525 


PLT 3 


0.16371 


PSVC 1 


0.321222 


COMP 1 


0.369528 


TIS 2 


0.370832 


COM 3 


0.383773 


DEPL 1 


0.68538 


AD 5 


0.883773 


Intercept 


0.888516 


DEPL 2 


0.891042 


DEPL 3 


0.90257 


AD 2 


0.907533 


AD 3 


0.908714 


AD 4 


0.909825 


AD 1 


0.919134 


COM 2 


0.949266 


TIS 3 


0.982466 
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The results from the LRM show that there is no attribute with a statistically 
significant impact (p value of <=.05). These results suggest that none of the seven 
demographic attributes significantly contributed to a participant performing above 
average when identifying hazards on a risk assessment. 

6.3 Analysis of results 

This study intended to answer the question, ‘Would the demographic attributes of 
the participants in this study have a significant impact on their overall ability to correctly 
identify the hazards presented in the scenarios provided?’, which lead to the following 
hypotheses: 

Ho: Demographic attributes do not have an impact on the overall performance of 
the participants. 

H,: Demographic attributes do have an impact on the overall performance of the 
participants. 

As illustrated in Chapter 6, the results of the research showed that none of the 
demographic attributes considered as part of the study had a significant statistical impact 
on a participant’s ability to achieve a “passing” score. The researcher therefore has to 
conclude that, given the hypotheses established at the outset of this study, the research 
failed to reject the null hypothesis but did not prove the alternative hypothesis. This 
means that while the results did not conclusively prove that experience or other 
demographic attributes had a statistically significant impact on a participant’s overall 
performance, the results do suggest that the idea that those same attributes do not have an 
impact cannot be rejected. 
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Considering the fact that the least Time in Service for any participant was 4 
months, while the greatest was 1 3 years, the conventional wisdom that the longer or more 
often an individual has been around a process, the better they will be at it does not appear 
to stand the rigors of statistical analysis, at least in this study’s case. Similarly, the notion 
that any prior service time at all would increase a participant’s performance could not be 
proven. Less than half of participants with prior enlisted time scored above average in 
both scenarios. Further studies would need to be done to prove this conclusively since 
this fact may merely suggest that their prior service was in a grade junior enough that 
they were never formally exposed to the RM process. 

Finally, it must be noted that the absence of a statistically significant correlation 
between the participants’ performance and the demographic attributes considered for 
each participant could be attributed to the fact that immediately prior to being provided 
the scenarios and RM products, each study participant received a standardized block of 
instruction on the U.S. Army RM process. This block of instruction, which includes 
instruction on using the METT-TC construct for identifying hazards (as was discussed in 
Chapter 1), may have served its intended purpose by mitigating any significant 
experiential differences that existed between the participants. 
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7. Discussion 

This study is as a result of over 18 years of the researcher learning, teaching, 
participating in, and observing the U.S. Army’s RM process in action. Throughout that 
time, the researcher has tried to gain an understanding of why different people perceive 
the presence or absence of hazards so differently. The conventional wisdom so often 
prevalent within the Army that experience is the key to success in RM may stand true, 
but that does not answer the question of what kind of experience? Prior service in the 
military? Experience gained while attending a particular commissioning source? The 
overwhelming majority of literature available today on the subject of RM acknowledges 
that there is inherent subjectivity in nearly every RM process in existence. But what if 
this subjectivity were based on experience or some other source that lent itself to a better 
outcome? Are there certain attributes or experiences that assist a person in accurately 
identifying hazards? That is what this study was intended to address. The study analyzed 
the results of two scenario-based risk assessments completed by a population of newly- 
commissioned Infantry Officers who were attending the U.S. Army Infantry Basic 
Officers Leadership Course at Fort Benning, GA. Input was analyzed using both a NN 
algorithm to produce DoS from an expert case for each respondent in each scenario. 
Those outputs then served as variables in a LRM intended to identify any of seven 
attributes that may have had a statistically significant impact on a respondent’s success. 
Ultimately the results proved that none of the seven demographic attributes considered as 
part of the study had a significant statistical impact on a respondent’s ability to achieve a 
“passing” score. As was discussed in Chapter 6, the researcher had to conclude that, 
given the hypotheses established at the outset of this research study, the research results 
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failed to reject the null hypothesis but did not prove the alternative hypothesis. This 
means that while the results did not conclusively prove that experience or other 
demographic attributes had a statistically significant impact on a respondent’s overall 
performance, the results do suggest that the idea that those same attributes do not have an 
impact cannot be rejected. 

7.1 Limitations of the study 

This study was constrained by three primary limitations. First was the relatively 
small sample size of 72 respondents. A larger sample size may have shown that one or 
more of the demographic attributes had a significant impact. Unfortunately, due to the 
time of year in which the study was conducted, the IBOLC class in session at the time of 
the study was smaller than normal. Additionally, the time of year also resulted in a class 
mix that was much heavier on service academy graduates than other timeframes would 
have been. This is due to the graduation dates of each of the commissioning sources. 
The U.S. Military Academy graduates in early May and those officers have priority for 
class seats in the early summer IBOLC classes. Lieutenants receiving their commission 
through ROTC typically fill the bulk of mid to late summer IBOLC classes, with OCS 
commissionees filling in throughout the year due to their cyclic graduation dates. This 
resulted in a disproportional number of service academy commissionees serving as 
participants. Finally, the actual instrument used for gathering data, the DA Form 7566, 
Composite Risk Management Worksheet, was designed to serve as a risk management 
tool, not a test instrument as it was in the study. As such, it was not designed using an 
approach based in a proven research methodology to ensure that it fully captures the 
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scope of anticipated learned knowledge or performance measures when used in a data 
collection role. Although it served its purpose in this study based on its familiarity to all 
respondents and its doctrinal foundation, future studies may consider an alternative set of 
test measures that are first validated as test instruments by checking for internal 
consistency. 

7.2 Recommendations for further work 

It must be noted that one potential reason for the lack of any statistically 
significant impact by the demographic might have occurred as a result of the RM training 
provided by the IBOLC instructors prior to the respondents completing their risk 
assessments. This standardized block of instruction may have had a mitigated effect on 
the disparity that could be caused by varying backgrounds, experiences, etc, and 
essentially caused a “leveling of the playing field” in the participants’ ability to identify 
hazards. If this were the case, it would seem to speak highly of the Army’s current RM 
Chain Teaching model used to instruct service members on RM. This study however, did 
not directly address this idea, and the researcher recommends this as a topic for further 
study utilizing a pre/post-test methodology. This study could analyze the impacts of the 
standardized instruction by recording the participants’ results from two scenarios 
completed prior to their receiving the standardized block of RM instruction. Once they 
have received the instruction, the researchers could present the participants with two 
more scenarios, similar in nature to the first two, and use the differences in performance 
data to determine what, if any, impacts the standardized block of RM instruction had on 
the population’s ability to correctly identify hazards. Finally, the researcher recommends 




40 



that a similar study can be conducted but with alternative demographic attributes such as 
respondent age, specific Military Occupational Specialty (MOS) if the respondent had 
served previously in the military, specific college major (versus academic discipline 
category such as the researcher used), etc, in order to determine if there might be a 
correlation between those attributes and how the participants react to different RM 
scenarios in the U.S. Army. 
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